Two-way interleaving in a three-rank environment

ABSTRACT

A memory controller maintains a mapping of target ranges in system memory space interleaved two-ways across locations in a three-rank environment. For each range of the target ranges, the mapping comprises a two-way interleaving of the range across two ranks of the three-rank environment and offsets from base locations in the two ranks. At least one of the ranges has offsets that differ relative to each other. Such offsets allow the three ranks to be fully interleaved, two ways. An instruction to read data at a rank-agnostic location in the diverse-offset range causes the memory controller to map the rank-agnostic location to two interleaved locations offset different amounts from their respective base locations in their ranks. The controller may then affect the transfer of the data at the two interleaved locations.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and benefit from U.S. patentapplication Ser. No. 16/874,998 titled “TWO-WAY INTERLEAVING IN ATHREE-RANK ENVIRONMENT” filed on May 15, 2020, the content of which isexpressly incorporated by reference in its entirety for all purposesherein.

TECHNICAL FIELD

Aspects of the disclosure are related to the field of computinghardware, software, and firmware, and in particular, to enhanced memoryinterleaving.

BACKGROUND

Memory interleaving is a technique by which contiguous data is staggeredacross multiple physical resources so as to increase bandwidth. Thephysical resources used to provide system memory in a computer require acertain amount of time between read/write operations. This fact causes abottleneck when contiguous data is stored sequentially on a singlephysical resource because the processor has to wait a period of timeafter accessing the resource before accessing it again. Interleavingsolves this problem by spreading out contiguous blocks of data acrosstwo or more physical resources so that the resources can be accessed ata speed that more closely matches that of the channels to and from theprocessor.

Ranks are the logical divisions of physical resources across which datamay be interleaved in a given environment, while n-way interleavingrefers to the number of ranks being interleaved in a configuration. Forexample, two-way interleaving describes a technique whereby contiguousblocks of data are interleaved across two ranks in a staggered manner,whereas four-way interleaving would interleave the data across fourranks. Two-way (or four-way) interleaving therefore scales-up very wellin environments that have a quantity of ranks equal to a power-of-twonumber (i.e. 2^(X)) such as those with 2 ranks, 4 ranks, 8 ranks, and soon.

Two-way interleaving benefits “power-of-two” rank environments becausethe rank-agnostic addresses in memory system space map seamlessly to therank-aware address space of the physical memory resources. As anexample, two contiguous locations in memory system space (e.g.rank-agnostic addresses j and k in a range) would map directly to thesame location on two different physical memory resources (e.g.rank-aware addresses α1 and α2, where 1 and 2 denote the differentphysical resources and α represents the location on each resource). Thisallows data to be interleaved by way of a simple mapping between addressspaces and it allows all of the system memory to be interleaved.

Two-way interleaving in three-rank environments (e.g. those with 3, 6,9, 12 or more physical memory resources) is less efficient because oneof three regions in memory space cannot be interleaved. To illustrate,assume that system memory is divided into three address ranges and thatthe physical memory resources include three ranks. Two-way interleavingwould map the first of three ranges to the first half of each of tworanks; the second range to the second half of each of the same tworanks; and the third range to the entirety of the third rank. In otherwords, the data in the third range of addresses would not beinterleaved, thereby limiting the effectiveness of two-way interleavingin a three-rank environment.

The limitations of two-way interleaving in three-rank environments hasthe practical consequence of increasing system costs unnecessarily. Takefor example a server configuration with three ranks of 16 GB memory or48 GB total. 32 GB of the 48 GB can be interleaved two ways, while theremaining 16 GB cannot be interleaved. The memory could be increased to64 GB, thereby allowing the entirety of the memory to be interleaved,but at the considerable expense of adding a fourth memory module.

BRIEF SUMMARY

Technology for performing two-way interleaving in a three-rankenvironment as disclosed herein eliminates or mitigates theinefficiencies of prior techniques. In various implementations, a memorycontroller loads a mapping of target ranges in system memory spaceinterleaved two-ways across locations in a three-rank environment. Foreach range of the target ranges, the mapping comprises a two-wayinterleaving of the range across two ranks of the three-rank environmentand offsets from base locations in the two ranks.

At least one of the ranges has offsets that differ relative to eachother, meaning that data allocated to that range is interleaved atdifferent positions in their respective ranks. Such non-uniform offsetsallow the three ranks to be fully interleaved, two ways. An instructionto read data at a rank-agnostic location in the diverse-offset rangecauses the memory controller to map the rank-agnostic location to twointerleaved locations offset different amounts from their respectivebase locations in their ranks. The controller may then affect thetransfer of the data at the two interleaved locations.

This Brief Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. It may be understood that this Overview is not intended toidentify key features or essential features of the claimed subjectmatter, nor is it intended to be used to limit the scope of the claimedsubject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Many aspects of the disclosure may be better understood with referenceto the following drawings. The components in the drawings are notnecessarily to scale, emphasis instead being placed upon clearlyillustrating the principles of the present disclosure. Moreover, in thedrawings, like reference numerals designate corresponding partsthroughout the several views. While several embodiments are described inconnection with these drawings, the disclosure is not limited to theembodiments disclosed herein. On the contrary, the intent is to coverall alternatives, modifications, and equivalents.

FIG. 1 illustrates an operational architecture in an implementation oftwo-way interleaving in three-rank environments.

FIG. 2 illustrates an interleaving process in an implementation.

FIG. 3 illustrates a supplemental view of the operational architectureof FIG. 1.

FIG. 4 illustrates an operational scenario in an implementation.

FIG. 5 illustrates several views of an operational environment in whichthe various architectures, processes, scenarios, and other aspects oftwo-way interleaving in three-rank environments discussed herein may beimplemented.

DETAILED DESCRIPTION

Techniques for performing two-way interleaving in three-rankenvironments are disclosed herein. In various implementations, a uniquemapping of system memory space to memory module space allows all of theranks in a three-rank environment to have interleaved data storedthereon. Such an advance allows computing devices (e.g. servercomputers) to be configured with 3n ranks of memory to better matchtheir workloads, while still benefiting from the advantages of two-wayinterleaving.

FIG. 1 illustrates an operational architecture 100 for implementingtwo-way interleaving in a three-rank environment. Operationalarchitecture 100 includes a controller 101 and three ranks of memoryrepresented by rank 121 (rank_a), rank 131 (rank_b), and rank 141(rank_c). Ranks 121, 131, and 141 are connected to bus 115, as iscontroller 101. Bus 115 is representative of one or more buses overwhich data and control signals flow to and from the memory representedby ranks 121, 131, and 141.

Ranks 121, 131, and 141 each comprises a set of memory devices thatoperate in unison, examples of which include dynamic random-accessmemory (DRAM) devices and any other type of memory device. Controller101 comprises a digital circuit that manages the flow of data going toand from the memory represented by ranks 121, 131, and 141. Controller101 may be integrated as part of a microprocessor (e.g. the centralprocessing unit) or may be implemented separately.

Controller 101 maintains a mapping 110 of rank-agnostic memory space torank-aware memory space. The rank-agnostic memory space comprises theaddress space understood by the central processor when requesting toread and write data. The rank-aware memory space comprises the addressspace that describes the actual physical locations on the memory deviceswhere read/write operations are affected. The parameters of mapping 110may be loaded into controller 101 from a basic input/output system(BIOS) upon power-up, although mapping 110 could be staticallymaintained in some implementations or provided from some other source.Mapping 110 may be a file, a list, and array, or any suitable datastructure. It may be appreciated that mapping 110 in FIG. 1 is merely anexemplary visualization to illustrate the relationship of memory-spaceaddresses to physical locations.

It is assumed for exemplary purposes that mapping 110 describes atwo-way interleaving of three ranges of addresses in memory space to thethree-ranks in operational architecture 100. The three ranges arerepresented by range 110 (range_0), range 111 (range_1), and range 112(range_2). Each range is illustrated with a different fill pattern thatdistinguishes one range from another. It may be appreciated that theconcepts disclosed herein apply as well to four-way interleaving. Inaddition, the concepts disclosed herein can be applied with respect tothree-rank environments having any 3n number of ranks (e.g. 3, 6, 9, 12,and so on).

Each range also includes two rank identifiers and their respectiveoffsets from their corresponding base address in the ranks. For example,data allocated to addresses in range 110 is interleaved between rank 121and rank 131. In addition, the offsets used to access ranks 121 and 131are the same: zero. Data allocated to addresses in range 112 isinterleaved between rank 121 and rank 131 and their offsets are also thesame with respect to each other: one-half the size of each rank.However, the offsets used to access rank 121 and rank 141 for purposesof interleaving data allocated to range 111 differ with respect to eachother. That is, the offset for rank 121 comprises one-half the size ofrank 121, whereas the offset for rank 141 comprises zero. The differentoffsets for the interleaving of data allocated to range_b make itpossible to fully interleave data in a two-way manner across the threeranks.

The same fill patterns used to distinguish each address range in mapping110 are also used to illustrate the logical subdivisions of each rankthat map to the different ranges. For example, a first subdivision 123of rank 121 maps to range 110, while a second subdivision 125 of rank121 maps to range 111. A first subdivision 133 of rank 131 maps to range110 and a second subdivision 135 of rank 131 maps to range 111. Lastly,a first subdivision 143 of rank 141 maps to range 111 and a secondsubdivision 145 maps to range 112. The first and section subdivision ofeach rank may be equally sized with respect to each other and withrespect to the other subdivision of the other ranks. Similarly, each ofthe ranges 110, 111, and 112 in memory space may be equally sized.

Controller 101 manages the flow of data in accordance with interleavingprocess 200. The logic of interleaving process 200, illustrated in FIG.2, may be implemented in the hardware and/or firmware of controller 101.The logic directs controller 101 to operate as described forinterleaving process 200, referring parenthetically to the steps in FIG.2.

In operation, controller 101 generates mapping 101 upon start-up (step201). Generating mapping 101 may include controller 101 obtainingparameters from BIOS or elsewhere that specify the type of interleaving(e.g. 2-way, 4-way) and the number of ranks (e.g. 3, 6, 8, etc.).Controller 101 then maps system memory space to physical memory spacebased on the type of interleaving and number of ranks specified in BIOS.This includes determining for each range of addresses in memory systemspace the ranks to which a given address range is mapped and the offsetsinto the ranks.

Controller 101 determines the offsets based at least partly on thecombination of interleaving type and the number of ranks. Two-way (orfour-way) interleaving in in a power-of-two number of ranks results inoffset pairs that match each other for each range of addresses mapped toa given pair of ranks. The offset pair for a given range need not be thesame as any other offset pair for any other range, although the offsetvalues within a given pair will match.

In contrast, two-way (or four-way) interleaving in a 3n-number of ranksresults in at least one range that maps to two ranks at differentoffsets with respect to each other. That is, the offset into one rankfor a given address range will differ from the offset into a secondrange for the same address range. It may be appreciated then thatcontroller 101 calculates the offsets differently depending upon thecombination of interleaving type and rank quantity. Such logic may beimplemented in hardware or firmware in the context of controller 101.

As an example, two-way interleaving across three, six, nine, or twelveranks (or other non-power of two quantities) would result in the sameoffsets into two-thirds of the rank pairs, but different offsets intoone-third of the rank pairs. The size of each rank may also factor intothe offset calculations. For instance, a one-half offset into 16 GBranks would start at 8 GB, while the same offset into 32 GB ranks wouldstart at 16 GB.

In other implementations, the parameters may specify the offsets asdiscrete items of information, in addition to the interleaving type. Forexample, the BIOS could initialize controller 101 with parametersindicative of the type of interleaving, the number of ranks, the size ofeach rank, and the offsets into the ranks. Alternatively, the BIOS couldinitialize controller 101 with the actual offset locations rather thanproviding the offset values. In either case, controller 101 may populatemapping 110 with an indication of the ranks mapped to each addressrange, as well as the offsets into the ranges, whether by offset valuesor locations.

In some implementations, controller 101 may calculate the offsetlocations a-priori using the offsets in the parameters, in which casemapping 110 could indicate a direct correspondence between addressranges and offset locations in the ranks. In other implementations,controller 101 may populate mapping 110 with the rank identifierscorresponding to the ranges, as well as the offsets as separate,discrete items of information. Controller 101 would then calculate theoffset locations at run-time after having accessed mapping 110. Othermechanisms for affecting the offsets are possible and may be consideredwithin the scope of the present disclosure.

Next, controller 101 receives an instruction to access a rank-agnosticlocation in system memory space (step 203). The instruction may compriseone or more suitable instructions such as load/store commands thatcorrespond to read/write operations and that cause data to be read fromor written to memory. The instruction identifies the location in memoryspace to be accessed for purposes of read/write operations and may beidentified by a single address, multiple addresses, a range ofaddresses, or the like.

Controller 101 responsively accesses its mapping 110 of rank-agnosticlocations to rank-aware locations to identify, based on the mapping, twointerleaved locations (in a 3-rank environment) corresponding to therank-agnostic location in memory space (step 205).

In some implementations, there may be a 1:1 correspondence betweenaddresses in system memory space and the addresses in memory modulespace before any interleaving is taken into consideration. For example,a first location in system memory space would correspond directly to asecond location in the physical layout of memory.

With interleaving, the correspondence between memory space and thephysical layout of memory locations becomes staggered. For example, afirst location in system memory space (e.g. a block of addresses) isinterleaved across two different ranks of physical memory. The simple1:1 correspondence is therefore lost. Rather, mapping 101 provides a mapof sorts for controller 101 to arrive at the interleaved physicallocations for a given location in system memory space. For example, inthe absence of interleaving, two consecutive addresses in system memoryspace would map to two consecutive addresses in physical memory. Withtwo-way interleaving in a two (or four) rank environment, two (or four)consecutive addresses in system memory space would map to a singlephysical address in physical memory, but at two (or four) differentranks, and so on for the remainder of addresses.

In the case of two-way interleaving in power-of-two rank environments,this relationship would hold for all of the addresses in system memoryspace. In the three-rank environments contemplated herein, offsets areutilized to sub-divide three ranks into six subdivisions of the threeranks. In addition, system memory space is sub-divided into threeranges. Each range of addresses in system memory space is allocated totwo different ranks and at an offset within each rank. Two-thirds of theranges have offsets into their respective ranks that are the same foreach rank in the pair, while one-third of the ranges have offsets intotheir respective rank pair(s) that differ. This allows the three rangesto be fully interleaved across the six subdivisions of the three ranksof memory. This provides an advance in speed and efficiency overprior-art techniques for two-way interleaving in three-rank environmentswhere two of three ranks were utilized fully for interleaving, but oneof three would contain no interleaved data.

Still referring to FIG. 2, controller 101 proceeds to cause the twointerleaved locations to be accessed such that data may be read from orwritten to the locations (step 207). This may be accomplished by, forexample, enabling and/or disabling control lines on bus 115 such thatdata can flow to or from the two interleaved locations.

FIG. 3 illustrates a supplemental view 300 of portions of operationalarchitecture 100 in an exemplary application of interleaving process200. In supplemental view 300, range 110 has been mapped to subdivision123 of rank 121 and subdivision 133 of rank 131 by virtue of thezero-offset indicated for both ranks. Range 111 has been mapped tosubdivision 125 of rank 121 and subdivision 143 of rank 141 by virtue ofthe one-half and zero offsets indicated respectively. Lastly, range 112has been mapped to subdivision 135 of rank 131 and subdivision 145 ofrank 141 by virtue of the one-half offsets indicate for both ranks.

FIG. 4 illustrates an operational scenario 400 to further demonstratevarious aspects of the enhanced interleaving proposed herein.Operational scenario 400 represents the two-way interleaving of dataallocated to system memory space 410 across three ranks of physicalmemory, represented by rank 421, 431, and 441. A mapping 420 controls ordirects the interleaving of three ranges of address space across sixsubdivisions of the three ranks. The three address ranges arerepresented by range 411, range 412, and range 413. The six subdivisionsare represented by subdivision 423 and subdivision 425 of rank 421,subdivision 433 and subdivision 435 of rank 431, and subdivision 443 andsubdivision 445 of rank 441.

The mapping 420 directs the data allocated to contiguous addresses insystem memory space 410 to be interleaved across subdivisions of tworanks per range of addresses. The data comprises contiguous datarepresented symbolically for exemplary purposes. Range 411 maps tosubdivision 423 of rank 421 and subdivision 433 of rank 431; range 412maps to subdivision 425 of rank 421 and subdivision 433 of rank 441; andrange 413 maps to subdivision 435 of rank 431 and subdivision 445 ofrank 441. It may be appreciated that the offsets into subdivisions 423and 433 are the same with respect to each other, the offsets intosubdivision 425 and 443 differ with respect to each other, and theoffsets into subdivisions 435 and 445 are the same with respect to eachother (although they differ relative to those of subdivision 423 andsubdivision 433).

FIG. 5 illustrates two views of an operational environment in animplementation of enhanced interleaving. View 501 presents a side viewof the operational environment, while view 502 presents a top-down view.The operational environment includes a controller 505 and various memorymodules represented by memory module 510, memory module 511, and memorymodule 512. Examples of memory modules 510-512 include dual in-linememory modules (DIMM), single in-line memory modules (SIMM), and anyother type of random-access memory (RAM) module. Each memory module mayinclude one or more independent ranks, and each rank comprises a set ofmemory devices (e.g. DRAM devices) that operate in unison, of whichmemory device 515 is representative. Internal to each memory device areone or more independent banks comprised of memory arrays.

Package pins connect the memory devices to printed circuit boardmaterial which is connected via edge connectors 507 to bus traces. View502 includes an example of a printed circuit board 520 that holds a set521 of memory devices. The printed circuit board 520 includes pins 523that establish a connection through edge connectors 507.

Controller 505 may be initialized during boot-up to perform two-wayinterleaving across 3n-number of ranks which, in this case, equals six.View 502 illustrates an arrangement of a 3n-number of ranks in theoperational environment. In view 502, rank_a is defined as the set ofmemory devices on one side of memory module 510, while rank_b is definedas the set of memory devices on the other side of memory module 510.Rank_c and rank_d are defined as the sets of memory devices on eitherside of memory module 510, while rank_e and rank_f are the sets ofmemory devices on either side of memory module 512 respectively.

Controller 505 employs an interleaving process (e.g. interleavingprocess 200) to efficiently write data to—and read data from—memorymodules 510-512. While six ranks are illustrated, it may be appreciatedthat the concepts described herein could be scaled down to three ranksor up to any 3n-numbers of ranks (e.g. 9, 12, and so on). The conceptsalso apply to four-way interleaving It may be further appreciated thatsingle in-line memory modules may be employed instead of—or in additionto—the dual in-line memory modules illustrated in FIG. 5.

The enhanced interleaving techniques disclosed herein may be implementedin the context of any computing device, examples of which include, butare not limited to, server computers, desktop computers, laptopcomputers, tablet computers, mobile phones, watches and other wearabledevices, and Internet of Things (IoT) devices.

An exemplary computing device includes, but is not limited to, aprocessing system, a storage system, software, a communication interfacesystem, and a user interface system (optional). The processing system isoperatively coupled with the storage system, the communication interfacesystem, and the user interface system. The memory controllers disclosedherein may be implemented in the context of the processing system (e.g.CPU) or separately within an exemplary computing device.

An exemplary memory controller includes processing circuitry andnon-transitory computer readable storage media readable by theprocessing circuitry and capable of storing firmware and associatedparameters. The computer readable storage media may include volatile andnonvolatile media, removable and non-removable media, or any othermethod or technology for storage of information, such as computerreadable instructions, data structures, program modules, or other data.Examples of the storage media include random access memory, read onlymemory, or any other suitable storage media. In no case is the computerreadable storage media a propagated signal.

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

The included descriptions and figures depict specific embodiments toteach those skilled in the art how to make and use the best mode. Forthe purpose of teaching inventive principles, some conventional aspectshave been simplified or omitted. Those skilled in the art willappreciate variations from these embodiments that fall within the scopeof the disclosure. Those skilled in the art will also appreciate thatthe features described above may be combined in various ways to formmultiple embodiments. As a result, the invention is not limited to thespecific embodiments described above, but only by the claims and theirequivalents.

What is claimed is:
 1. A computing apparatus comprising: a three-rankenvironment comprising a plurality of memory modules, wherein theplurality of memory modules comprises a quantity of memory modules equalto a non-power of two number; a memory controller operatively coupledwith the plurality of memory modules and configured to at least: load amapping of target ranges in system memory space interleaved two-waysacross locations in the three-rank environment, wherein the targetranges comprise three ranges of rank-agnostic addresses for at least arange of the target ranges, the mapping comprises a two-way interleavingof the range across two ranks of the three-rank environment and offsetsfrom base locations in the two ranks, and the offsets differ relative toeach other for at least one range of the three ranges of rank-agnosticaddresses; receive an instruction to read data at a rank-agnosticlocation in the system memory space; using the mapping, map therank-agnostic location to two interleaved locations offset differentamounts from the base location in each of the two ranks mapped to therange of the rank-agnostic location; and read the data at the twointerleaved locations.
 2. The computing apparatus of claim 1 wherein thenon-power of two number comprises one of three, six, nine, and twelve.3. The computing apparatus of claim 1 wherein the offsets match eachother for two ranges of the three ranges of rank-agnostic addresses. 4.The computing apparatus of claim 3 wherein the rank-agnostic locationcomprises a rank-agnostic address in the system memory space.
 5. Thecomputing apparatus of claim 4 wherein the two interleaved locationscomprise rank-aware addresses in memory module space.
 6. The computingapparatus of claim 1 wherein the plurality of memory modules comprisesdynamic random-access memory (DRAM).
 7. The computing apparatus of claim1 wherein the three ranges of rank-agnostic addresses are sized equallywith respect to each other.
 8. A method of operating a memory controlunit to perform two-way interleaving in a three-rank environment, themethod comprising: loading a mapping of target ranges in system memoryspace interleaved two-ways across locations in the three-rankenvironment, wherein: the target ranges comprise three ranges ofrank-agnostic addresses; for at least a range of the target ranges, themapping comprises a two-way interleaving of the range across two ranksof the three-rank environment and offsets from base locations in the tworanks, and the offsets differ relative to each other for at least onerange of the three ranges of rank-agnostic addresses; and the three-rankenvironment includes a plurality of memory modules equal to a non-powerof two number; receiving an instruction to read data at a rank-agnosticlocation in the system memory space; using the mapping, mapping therank-agnostic location to two interleaved locations offset differentamounts from the base location in each of the two ranks mapped to therange of the rank-agnostic location; and reading the data at the twointerleaved locations.
 9. The method of claim 8 wherein the non-power oftwo number comprises one of three, six, nine, and twelve.
 10. The methodof claim 8 wherein the offsets match each other for two ranges of thethree ranges of rank-agnostic addresses.
 11. The method of claim 10wherein the rank-agnostic location comprises a rank-agnostic address inthe system memory space.
 12. The method of claim 11 wherein the twointerleaved locations comprise rank-aware addresses in memory modulespace.
 13. The method of claim 8 wherein the plurality of memory modulescomprises dynamic random-access memory (DRAM).
 14. The method of claim 8wherein the three ranges of rank-agnostic addresses are sized equallywith respect to each other.
 15. A computing apparatus comprising: amemory sub-system comprising a three-rank environment that includes aplurality of memory modules equal to a non-power of two number; one ormore processors configured to read and write data to and from the memorysub-system; and a basic input/output system configured to initialize thecomputing apparatus to perform two-way interleaving of the data acrossthe three-rank environment by: loading a mapping of target ranges insystem memory space interleaved two-ways across locations in thethree-rank environment, wherein: the target ranges comprise three rangesof rank-agnostic addresses, for at least a range of the target ranges,the mapping comprises a two-way interleaving of the range across tworanks of the three-rank environment and offsets from base locations inthe two ranks, and the offsets differ relative to each other for atleast one range of the three ranges of rank-agnostic addresses;receiving an instruction to read the data at a rank-agnostic location inthe system memory space; using the mapping, mapping the rank-agnosticlocation to two interleaved locations offset different amounts from thebase location in each of the two ranks mapped to the range of therank-agnostic location; and reading the data at the two interleavedlocations.
 16. The computing apparatus of claim 15 wherein the non-powerof two number comprises one of three, six, nine, and twelve.
 17. Thecomputing apparatus of claim 15 wherein the offsets match each other fortwo ranges of the three ranges of rank-agnostic addresses.
 18. Thecomputing apparatus of claim 17 wherein the rank-agnostic locationcomprises a rank-agnostic address in the system memory space.
 19. Thecomputing apparatus of claim 18 wherein the two interleaved locationscomprise rank-aware addresses in memory module space.
 20. The computingapparatus of claim 15 wherein the plurality of memory modules comprisesdynamic random-access memory (DRAM).